Speech endpointing based on word comparisons

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for speech endpointing based on word comparisons are described. In one aspect, a method includes the actions of obtaining a transcription of an utterance. The actions further include determining, as a first value, a quantity of text samples in a collection of text samples that (i) include terms that match the transcription, and (ii) do not include any additional terms. The actions further include determining, as a second value, a quantity of text samples in the collection of text samples that (i) include terms that match the transcription, and (ii) include one or more additional terms. The actions further include classifying the utterance as a likely incomplete utterance or not a likely incomplete utterance based at least on comparing the first value and the second value.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. patent application is a continuation of, and claims priorityunder 35 U.S.C. § 120 from, U.S. patent application Ser. No. 16/540,795,filed on Aug. 14, 2019, which is a continuation of U.S. patentapplication Ser. No. 16/154,875, filed on Oct. 9, 2018, which is acontinuation of U.S. patent application Ser. No. 15/156,478, filed onMay 17, 2016, which is a continuation of U.S. patent application Ser.No. 14/681,203, filed on Apr. 8, 2015, which claims priority under 35U.S.C. § 119(e) to U.S. Provisional Application 61/983,025, filed onApr. 23, 2014. The disclosures of these prior applications areconsidered part of the disclosure of this application and are herebyincorporated by reference in their entireties.

TECHNICAL FIELD

This disclosure generally relates to speech recognition, and oneparticular implementation relates to endpointing speech.

BACKGROUND

Natural language processing systems typically use endpointers todetermine when a user has started and finished speaking. Sometraditional endpointers evaluate the duration of pauses between words indetermining when an utterance begins or ends. For instance, if a usersays “what is <long pause> for dinner,” a traditional endpointer maysegment the voice input at the long pause, and may instruct the naturallanguage processing system to attempt to process the incomplete phrase“what is,” instead of the complete phrase “what is for dinner.” If anendpointer designates an incorrect beginning or ending point for a voiceinput, the result of processing the voice input using the naturallanguage processing system may be inaccurate or undesirable.

SUMMARY

According to an innovative aspect of the subject matter described inthis specification, a computing device may receive an audio input of anutterance spoken by a user, and may use a continuous speech recognizerto incrementally transcribe the utterance. The computing device comparesthe incrementally recognized transcription to text samples from acollection of text samples, such as a collection of search queries thatwere previously submitted to a search engine by other users, todetermine whether the transcription more likely represents a completequery or an incomplete query.

According to one approach, determining whether the transcription morelikely represents a complete query or an incomplete query includesdetermining a quantity of the text samples that match the transcriptionand that do not include any additional terms, and determining a quantityof text samples that match with the transcription and that do includeone or more additional terms. The computing device may then determine aratio based on those two quantities and compares the determined ratio toa threshold ratio. If the determined ratio does not satisfy thethreshold ratio, then the computing device classifies the utterance aslikely an incomplete utterance. If the determined ratio satisfies thethreshold ratio, then the computing device classifies the utterance asnot likely an incomplete utterance.

The device may endpoint the voice input, or may deactivate themicrophone or maintain the microphone in an active state based on theclassification of the utterance as likely a complete utterance or anincomplete utterance. If the utterance is classified as likely anincomplete utterance, then the device can maintain the microphone in anactive state to receive an additional utterance, or may wait furtherbefore endpointing the voice input. If the utterance is classified asnot likely an incomplete utterance, then the device can deactivate themicrophone and process the utterance, or may not wait further beforeendpointing the voice input.

In general, another innovative aspect of the subject matter described inthis specification may be embodied in methods that include the actionsof obtaining a transcription of an utterance; determining, as a firstvalue, a quantity of text samples in a collection of text samples that(i) include terms that match the transcription, and (ii) do not includeany additional terms; determining, as a second value, a quantity of textsamples in the collection of text samples that (i) include terms thatmatch the transcription, and (ii) include one or more additional terms;and classifying the utterance as a likely incomplete utterance or not alikely incomplete utterance based at least on comparing the first valueand the second value.

These and other embodiments can each optionally include one or more ofthe following features. The actions of determining, as a first value, aquantity of text samples in a collection of text samples that (i)include terms that match the transcription, and (ii) do not include anyadditional terms includes determining that, in each text sample, thatterms that match the transcription occur in a same order as in thetranscription. The action of determining, as a second value, a quantityof text samples in the collection of text samples that (i) include termsthat match the transcription, and (ii) include one or more additionalterms includes determining that, in each text sample, the terms thatmatch the transcription occur at a prefix of each text sample.

The action of classifying the utterance as a likely incomplete utteranceor not a likely incomplete utterance based at least on comparing thefirst value and the second value includes determining a ratio of thefirst value to the second value; determining that the ratio satisfies athreshold ratio; and based on determining that the ratio satisfies thethreshold ratio, classifying the utterance as a likely incompleteutterance. The action of classifying the utterance as a likelyincomplete utterance or not a likely incomplete utterance based at leaston comparing the first value and the second value includes classifyingthe utterance as a likely incomplete utterance; and based on classifyingthe utterance as a likely incomplete utterance, maintaining a microphonein an active state to receive an additional utterance.

The action of classifying the utterance as a likely incomplete utteranceor not a likely incomplete utterance based at least on comparing thefirst value and the second value includes classifying the utterance asnot a likely incomplete utterance; and based on classifying theutterance as not a likely incomplete utterance, deactivating amicrophone. The actions further include receiving data indicating thatthe utterance is complete; where classifying the utterance as a likelyincomplete utterance or not a likely incomplete utterance based at leaston comparing the first value and the second value includes classifyingthe utterance as a likely incomplete utterance; and based on classifyingthe utterance as a likely incomplete utterance, overriding the dataindicating that the utterance is complete.

Other embodiments of this aspect include corresponding systems,apparatus, and computer programs recorded on computer storage devices,each configured to perform the operations of the methods.

In general, another innovative aspect of the subject matter described inthis specification may be embodied in methods that include the actionsof obtaining a transcription of an utterance; determining whether, in acollection of text samples, the utterance more often matches (i) textsamples that include terms that match the transcription and that do notinclude any additional terms than the utterance matches, or (ii) textsamples that include terms that match the transcription and that doinclude one or more additional terms than the utterance matches; anddetermining whether the utterance is likely incomplete based on thedetermination.

Other embodiments of this aspect include corresponding systems,apparatus, and computer programs recorded on computer storage devices,each configured to perform the operations of the methods.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. A user may use the voice input capabilities of acomputing device and speak at a pace that is comfortable for the user.An utterance may be endpointed at the intended end of the utterance,leading to more accurate or desirable natural language processingoutputs, and to faster processing by the natural language processingsystem. In the presence of background noise, an utterance may beendpointed at the intended end of the utterance.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A to 1C are diagrams of utterances and example signals used inclassifying whether the utterances are likely incomplete.

FIG. 2 is a diagram of an example system that classifies an utterance aslikely complete or likely incomplete.

FIG. 3 is a diagram of an example process for classifying whether anutterance is likely incomplete.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1A is a diagram 100 of an utterance and example signals used inclassifying whether the utterance is likely incomplete. In general,diagram 100 illustrates signals 102-108 that are generated or detectedby computing device 110 when the computing device 110 is processing anincoming audio input 102. The computing device 110 receives the audioinput 102 through a microphone or other audio input device of thecomputing device, and classifies whether the audio input 102 is anutterance that is likely incomplete or not likely incomplete.

The computing device 110 receives the audio input 102 and samples theaudio input 102 at a pre-designated frequency and resolution. Forexample, the computing device 110 may sample the audio input 102 at 8kHz, 16 kHz, 44.1 kHz, or any other sample rate, and the resolution maybe 16 bits, 32 bits, or any other resolution. Audio input 102illustrates sampled analog data that is based on an utterance 112 fromuser 114. In the example illustrated in FIG. 1A, user 114 says to thecomputing device 110, “What is . . . the meaning of life?” The computingdevice 110 may record and store the audio input 102 corresponding to theutterance 112.

The computing device 110 transcribes the utterance 112 spoken by theuser 114. In some implementations, the computing device 110 transcribesthe words of the utterance 112 using a processor of the computing device110 that is running automated speech recognition (ASR) software. Forexample, the computing device 110 may determine locally on the computingdevice 110 that an initial portion audio input 102 contains the term 120“what.” The ASR software receives the audio input 102 as the computingdevice 110 receives the utterance 112 from the user. As the ASR softwarerecognizes a term in the audio input 102, then the ASR software willprovide that term for output to the computing device 110. The computingdevice 110 records the amount of time between the ASR software returningterms that the ASR software recognizes. For example, the ASR softwaremay return the term 120 “what,” then one hundred millisecond laterreturn the term 122 “is,” and then two hundred milliseconds later returnthe term 124 “the.” The computing device 110 records that term 122 “is”follows term 120 “what” after one hundred milliseconds and term 124“the” follows term 122 “is” after two hundred milliseconds. In someimplementations, a server accessible through a network may be runningthe ASR software. In this case, the computing device transmits the audioinput 102 to the server over the network, receives terms from theserver, and records the amount of time between the server returningterms.

As the computing device 110 is transcribing the terms 120-130 of theutterance 112, the computing device 110 receives a general endpointersignal 104 from a general endpointer. The computing device 110 providesthe general endpointer with the terms received from the ASR software andthe recorded times between each term. The general endpointer comparesthe recorded times to a threshold. If one of recorded times between twoterms satisfies a threshold, then the general endpointer identifies anendpoint to be between the two terms. For example, the threshold may beone hundred fifty milliseconds. The general endpointer compares the timeof one hundred milliseconds between term 120 “what” and term 122 “is.”Because one hundred milliseconds is less than the threshold of onehundred fifty milliseconds, the general endpointer does not add anendpoint between term 120 “what” and term 122 “is.” The generalendpointer also compares the time of two hundred milliseconds betweenterm 122 “is” and term 124 “the.” Because two hundred milliseconds isgreater than the threshold of one hundred fifty milliseconds, thegeneral endpointer adds an endpoint between term 122 “is” and term 124“the.” The general endpointer signal 104 illustrates the endpoints astransitions between an active signal and an inactive signal. The generalendpointer signal is active between term 120 “what” and term 122 “is”and becomes inactive between term 122 “is” and term 124 “the.”

In some implementations, the general endpointer waits for a particularperiod of time after the end of a term before the general endpointeridentifies an endpoint. The general endpointer waits after each term isreceived from the ASR software or after the general endpointer receivedan indication that the ASR identified a term. If the waiting timesatisfies a threshold, then the general endpointer identifies anendpoint. For example, the threshold may be one hundred fiftymilliseconds. If the general endpointer receives an indication that theASR software has identified a term, such as term 120 “what,” then thegeneral endpointer will begin a timer. One hundred milliseconds later,the general endpointer receives an indication that the ASR software hasidentified another term, such as term 122 “is.” Because the timer of thegeneral endpointer did not reach the threshold of one hundred fiftymilliseconds, the general endpointer did not identify an endpoint afterterm 120 “what.” As another example, the general endpointer receives anindication that the ASR software has identified a term, such as term 122“is” and the general endpointer begins a timer. If one hundred fiftymillisecond later, the general endpointer has not received an indicationof a subsequent term, then the general endpointer identifies an endpointafter the term 122 “is.”

The computing device 110 classifies, using the terms that the ASRsoftware identifies from the audio input 102, the terms as a likelyincomplete utterance or not a likely incomplete utterance. The likelyincomplete utterance signal 106 illustrates the result of thisclassification. To classify the identified terms of the utterance 112 asa likely incomplete utterance, the computing device 110 compares theidentified terms to text samples in a collection of text samples. Thecomputing device 110 determines the quantity of text samples in thecollection of text samples that begin with the identified terms of theutterance 112 and contain additional terms. In other words, thecomputing device 110 determines the quantity of text samples in thecollection of text samples that contain a prefix that matches theidentified terms of the utterance 112. For example, the ASR softwareidentifies the first two terms of utterance 112 as “what is.” Thecomputing device 110 determines that of the collection of text samples,two thousand text samples begin with the prefix “what is” and containadditional terms. The computing device 110 also identifies the quantityof text samples in the collection of text samples that match theidentified terms of the utterance 112 and do not contain any additionalterms. For example, the computing device 110 determines that of thecollection of text samples, five contain the terms “what is” and noadditional terms.

With the quantity of text samples in the collection of text samples thatbegin with the identified terms of the utterance 112 and the quantity oftext samples that contain the identified terms of the utterance 112, thecomputing device 110 compares a ratio of the two quantities to athreshold ratio. The ratio represents a comparison of the quantity oftext samples that contain the identified terms and no other terms andthe quantity of text samples that begin with the identified and containadditional terms. If the ratio is less than the threshold ratio, thenthe computing device 110 classifies the identified terms as representinga likely incomplete utterance. If the ratio is more than the thresholdratio, then the computing device 110 classifies the identified terms asnot representing a likely incomplete utterance. For example, if theratio is 5:2000 and the threshold ratio is 1:200, then the computingdevice 110 would classify the identified terms as representing a likelyincomplete utterance.

In some implementations, the computing device 110 may invalidate thecomparison in instances where one or both of the quantities does notsatisfy a threshold. For example, if the ratio of the quantity of textsamples that contain the identified terms and no other terms and thequantity of text samples that begin with the identified and containadditional terms is 1:2, then the computing device 110 may invalidatethe comparison based on the quantity of text samples that begin with theidentified and contain additional terms being below a threshold of five.As another example, if the ratio of the quantity of text samples thatcontain the identified terms and no other terms and the quantity of textsamples that begin with the identified and contain additional terms is11,000:12,000, then then the computing device 110 may invalidate thecomparison based on the quantity of text samples that contain theidentified terms and no other terms or the quantity of text samples thatbegin with the identified and contain additional terms or both as beingabove a threshold of ten thousand as may be the case with a term such as“so.”

In some implementations, the computing device 110 does not compare theidentified terms to the collection of text samples unless the generalendpointer indicates that there is an endpoint between two terms. Forexample, the computing device 110 may not compare the term 120 “what” tothe collection of text samples because the general endpointer signal 104is active after the term 120 “what.” As another example, the computingdevice may compare the terms 120-122 “what is” to the collection of textsamples because the general endpointer signal is inactive after term122.

In some implementations, the computing device 110 does not compare theidentified terms to the collection of text samples unless the pausebetween terms satisfies a threshold. For example, the computing device110 may compare the identified terms if the pause between terms isgreater than one hundred fifty milliseconds. If the user 114 pauses onehundred milliseconds between term 120, “what,” and term 122, “is,” thenthe computing device 100 would not compare term 120, “what,” to thecollection of text samples. If the user 114 pauses two hundredmilliseconds between term 122, “is,” and term 124, “the,” then thecomputing device compares “what is” to the collection of text samples.

Based on the comparisons between the ratios of the quantity of textsamples that contain the identified terms and no other terms and thequantity of text samples that begin with the identified and containadditional terms and the threshold ratio, the computing device 110identifies that the utterance is likely incomplete until after the term130, “life.” In response to the general endpointer signal 104 beinginactive, the computing device 110 may have compared the ratio of thequantities to the threshold ratio with the terms 120 and 122 “what is”and determined that “what is” represents a likely incomplete utterance.In response to the general endpointer signal 104 being inactive, thecomputing device 110 may have compared the ratio of the quantities tothe threshold ratio with the terms 120-130 “what is the meaning of life”and determined that “what is the meaning of life” does not represent alikely incomplete utterance. The computing device 110 may havedetermined that terms 120-130 correspond to the beginning of one hundredtext samples and determined that the terms 120-130 correspond to onethousand full samples. The ratio would be 1000:100, which is greaterthan the threshold ratio of 1:200. Therefore, the computing device 110classifies the terms 120-130 as not likely an incomplete utterance. Asshown in FIG. 1A, the utterance 112 is likely incomplete until after theterm 130, “life.”

The computing device 112 concludes that the utterance 112 is complete asillustrated in the end of utterance signal 108. Once the computingdevice 112 concludes that the utterance 112 is complete, the computingdevice 110 outputs the utterance 112 as transcription 135. For example,the computing device 110 may output “what is the meaning of life” as thetranscription 135 of utterance 112.

In some implementations, the computing device 110 overrides a generalendpointing determination based on classifying an utterance is likelyincomplete. For example, the computing device 110 classifies “what is”as likely an incomplete utterance. The computing device 110 overridesthe general endpointing determination before determining that the end ofthe utterance is reached.

FIG. 1B is a diagram 150 of an utterance and example signals used inclassifying whether the utterances are likely incomplete. Diagram 150illustrates the computing device 160 processing the audio input 152 thatcorresponds to the utterance 162 “who is Barack Obama” spoken by user164. Similar to computing device 110 in FIG. 1A, computing device 160receives a general endpointer signal 154 for the audio input 152. Thecomputing device 160 receives the general endpointer signal 154 from ageneral endpointer, where the general endpointer identifies endpointsbased on the time between terms 120-126. For example, the computingdevice 160 determines that the time between term 170, “who,” and term172, “is,” is fifty milliseconds and determines that the time betweenterm 172, “is,” and term 174, “Barack,” is two hundred milliseconds. Ifthe threshold used by general endpointer for general endpointing is onehundred fifty milliseconds, then the general endpointer identifiesendpoints between term 172, “is,” and term 174, “Barack,” and noendpoints between term 170, “who,” and term 172, “is.” The generalendpointer signal 154 illustrates these endpoints before terms 170 and174 and after terms 172 and 176.

The computing device 160 generates a likely incomplete utterance signal156 as the ASR software identifies terms of the utterance 162 and thegeneral endpointer identifies endpoints. Similar to computing device 110in FIG. 1A, the computing device 160 determines the quantity of textsamples in a collection of text samples that match the identified termsand the quantity of text samples in the collection of text samples thatbegin with the identified terms and contain additional terms. Forexample, the computing device 160 determines that terms 170 and 172,“who is,” match fifty of the text samples in the collection of textsamples. To illustrate, the collection of text samples may contain fiftysamples of term 128, “whois.” The computing device 160 determines thatterms 170 and 172, “who is,” match the beginning of three thousand textsamples in the collection of text samples. To illustrate, the collectionof text samples that may contain samples such as “who is the pope,” “whois my congressman,” and “who is Barack Obama.”

The computing device 160 computes a ratio of the quantity of textsamples in a collection of text samples that match the identified termsto the quantity of text samples in the collection of text samples thatbegin with the identified terms. The computing device compares this to athreshold ratio to determine if the identified terms represent a likelyincomplete query. In this example, the ratio is 50:3000. As describedabove, the threshold ratio may be 1:200. As the ratio of 50:3000 isgreater than the threshold ration of 1:200, the computing device 160determines that the terms 170 and 172 are not a likely incompleteutterance as illustrated by term 178, “whois” and a drop in the likelyincomplete utterance signal 156 at the end of the audio input 152 atterm 178.

The computing device 160 determines that the end of the utterance hasbeen reached once the computing device 160 no longer classifies theidentified terms as likely an incomplete utterance. The end of utterancesignal 158 indicates that the computing device 160 has identified an endof utterance at the audio input 152 at term 178. The computing device160 outputs the term 178 as utterance 185.

In some implementations, the computing device 160 deactivates amicrophone of the computing device 160 based on classifying that theidentified terms are not likely incomplete. For example, the computingdevice 160 may deactivate a microphone used to receive the audio input152 once the computing device 160 determines the term 178, “whois” isnot a likely incomplete utterance. In some implementations, thecomputing device 160 may add an endpoint to the identified terms oncethe computing device identifies the terms as not likely incomplete. Forexample, the computing device may endpoint the utterance after the term178 “whois.”

In some implementations, the computing device may deactivate themicrophone based on the ratio of the quantity of text samples in acollection of text samples that match the identified terms to thequantity of text samples in the collection of text samples that beginwith the identified terms. The computing device may deactivate themicrophone if the ratio does not satisfy a particular range, leave themicrophone activated for a particular period of time if the ratiosatisfies the particular range, or leave the microphone activated untilthe computing device determines again that the utterance is likelyincomplete if the ratio does not satisfy the particular range. Forexample, the particular range of ratios may be 1:200 to 1:30. If thedetermined ratio is 5:2000, which is less than 1:200, then the computingdevice determines to leave the microphone activated. If the determinedratio is 5:300, which is between 1:200 and 1:30, then the computingdevice leaving the microphone activated for a particular time periodsuch as two seconds. If the computing device receives additional audioinput before two seconds, then the computing device processes theadditional audio input. If the computing device does not receiveadditional audio input within the two seconds, then the computing devicedeactivates the microphone. If the determined ratio is 1:20, then thecomputing device deactivates the microphone.

In FIG. 1B, the computing device 160 determines at a second point thatthe utterance is not likely incomplete. The computing device 160determines that the terms 174 and 176, “Barack Obama,” correspond to nota likely incomplete utterance, and therefore the computing device 160determines the end of the utterance after the term 176, “Obama,” asillustrated with the end of utterance signal 158. The computing device160 outputs two transcriptions of transcription 185, “whois,” andtranscription 188, “Barack Obama.”

FIG. 1C is diagram 200 of an utterance and example signals used inclassifying the utterance as likely incomplete or not likely incomplete.In general diagram 200 illustrates signals 202-208 that are generated ordetected by computing device 210 when the computing device is processingan incoming audio signal in the presence of background noise. Thecomputing device 210 receives an audio input 202 and background noisethrough a microphone or other audio input device of the computing device210 and classifies the audio input 202 as an utterance that is likelyincomplete or not likely incomplete. In the example illustrated indiagram 200, the background noise is music 218.

The example illustrated in FIG. 1C is similar to the example illustratedis FIG. 1A except for the additional music 218 that the computing device210 receives. In FIG. 1C, the use 214 speaks the utterance 212, “what is. . . the meaning of life.” The computing device 212 receives the audioinput 202 and music 218 and generates the general endpointing signal204. The initial endpoint for the audio input 202 corresponds to theterm 220, “what.” With the music 218 in the background, the computingdevice 212 is unable to identify an additional endpoint for the audioinput 202 because the signal strength of the audio input 202 does notfall below a threshold. As is illustrated by the general endpointingsignal 204 remains active even after the user 214 finishes speaking theutterance 212.

Despite the computing device 210 or a general endpointer being executedon the computing device 210 being unable to generate the generalendpointing signal 204 in the presence of the music 218, the computingdevice 210 identifies the terms 220-230 of the utterance 212. As above,the computing device 210 compares the identified terms to a collectionof text samples. The computing device 210 calculates the quantity oftext samples in the collection of text samples that begin with theidentified terms of the utterance 212 and the quantity of text samplesthat contain the identified terms of the utterance 212. The computingdevice 210 determines that the ratio is 5:2000 with the terms 220-222,“what is,” and that the ratio is 1000:100 with the terms 220-230, “whatis the meaning of life.” The computing device 210 compares the terms tothe threshold of 1:200 and determines that the terms 220-230, “what isthe meaning of life,” do not correspond to a likely incomplete utteranceas illustrated by the likely incomplete utterance signal 206. Thecomputing device 210 concludes that the utterance 212 has reached theend of the utterance based on the terms 220-230, “what is the meaning oflife,” not corresponding to a likely incomplete utterance. The computingdevice 210 then outputs the transcription 235, “what is the meaning oflife.”

FIG. 2 is a diagram of an example system 300 that classifies anutterance as likely complete or likely incomplete. The components of thesystem 300 may be contained in a computing device such as computingdevice 110, 160, or 210. The system 300 includes an audio subsystem 305.The audio subsystem 305 may receive an audio signal 310 from an outsidesource such as human speech. The audio subsystem 305 include amicrophone 315 to receive the audio signal 310. The audio subsystem 305converts the audio received through the microphone 315 to a digitalsignal using the analog-to-digital converter 320. The audio subsystem305 also includes buffers 325. The buffers 325 may store the digitizedaudio, for example, in preparation for further processing by the system300.

The enhanced end of utterance detector 330 receives and processes thedigitized audio from the audio subsystem 305. The enhanced end ofutterance detector 330 classifies the digitized audio as likely anincomplete utterance or not likely an incomplete utterance byidentifying terms of the digitized audio using the continuous speechrecognizer 350 and comparing the identified terms to a text corpus 345.Additionally, the enhanced end of utterance detector 330 identifiesgeneral endpoints for the digitized audio using the general endpointer340.

As the enhanced end of utterance detector 330 receives digitized audio,the enhanced end of utterance detector 330 uses the continuous speechrecognizer 350 to identify terms that correspond to the digitized audio.The continuous speech recognizer 350 uses automated speech recognitionto identify the terms that correspond to the digitized audio. In someimplementations, the continuous speech recognizer 350 may be implementedlocally on the device that receives the audio signal 310. In someimplementations, the continuous speech recognizer 350 is implemented ona server and the device that receives the audio signal 310 transmits thedigitized audio to the server over a network. The server then sends backterms of the digitized audio as the server identifies them. The enhancedend of utterance detector 330 records the time between receiving termsfrom the continuous speech recognizer 350. For example, the continuousspeech recognizer 350 identifies a term “what,” transmits the term tothe continuous speech recognizer 350, and one hundred milliseconds lateridentifies a term “is” and transmits the term to the continuous speechrecognizer 350.

As the enhanced end of utterance detector 330 receives digitized audio,the enhanced end of utterance detector 330 uses data from the generalendpointer 340 to identify general endpoints that correspond to thedigitized audio. In some implementations, the general endpointer 340analyzes the energy level of the digitized audio to determine endpoints.The general endpointer 340 may endpoint the digitized audio if theenergy level drops below a threshold for at least a particular amount oftime. For example, if the digitized audio corresponds to the userspeaking “what is” and then silence for three hundred milliseconds, thenthe general endpointer 340 determines that the silence is below anenergy threshold and the period of the silence, three hundredmilliseconds, is at least longer than a threshold time of two hundredmilliseconds. In this instance, the general endpointer 340 determinesthat there is an endpoint at the end of the digitized audiocorresponding to “is.”

In some implementations, the general endpointer 340 receives termsidentified by the continuous speech recognizer 350 and times betweeneach of the terms. If the time between two terms satisfies a threshold,then the general endpointer 340 adds an endpoint between those twoterms. For example, if between “what” and “is” is a period of onehundred milliseconds and the threshold is one hundred fiftymilliseconds, then the general endpointer may not add an endpointbetween “what” and “is” because the time period is below the threshold.As another example, if between “is” and “the” is a period of two hundredmilliseconds and the threshold is one hundred fifty milliseconds, thenthe general endpointer may add an endpoint between “is” and “the”because the time period is above the threshold.

The general endpointer 340 may also identify a general endpoint when theenergy level of the digitized audio increases above a threshold. Forexample, if the digitized audio corresponds to silence being followed bya user speaking “the,” then the general endpointer 340 may determinethat the energy of the digitized audio that corresponded to “the” isabove a threshold. In this instance, the general endpointer 340determines that there is an endpoint in the digitized audio thatcorresponds to the beginning of “the.”

The enhanced end of utterance detector 330 classifies the utterance thatcorresponds to the audio signal 310 as likely an incomplete utterance ornot likely an incomplete utterance using the prefix or completeutterance calculator 335. In some implementations, as the enhanced endof utterance detector 330 receives identified terms from the continuousspeech recognizer 350, the enhanced end of utterance detector 330compares those identified terms to the text corpus 345. The text corpus345 may include search queries performed by multiple users or searchqueries performed by a particular user. In some implementations, theenhanced end of utterance detector 330 determines the number of textsamples in the text corpus 345 that match the identified terms and thenumber of text samples in the text corpus 345 that begin with theidentified terms and do not contain additional terms. For example, ifthe identified term is “what,” then the enhanced end of utterancedetector 330 determines that “what” matches three of the text samples inthe text corpus 345 and “what” is the prefix of three thousand textsamples in the text corpus 345. If the identified terms are “what is,”then the enhanced end of utterance detector 330 determines that “whatis” matches three of the text samples in the text corpus 345 and “whatis” is the prefix of two thousand text samples in the text corpus 345.

In some implementations, the enhanced end of utterance detector 330determines the number of text samples in the text corpus 345 that matchthe identified terms when the identified terms are followed by a pausethat satisfies a threshold. For example, the threshold may be onehundred fifty milliseconds. If the identified term is “what” and isfollowed by fifty milliseconds, then the enhanced end of utterancedetector 330 may not determines the number of text samples in the textcorpus 345 that match the “what” and the number of text samples in thetext corpus 345 that begin with “what.” As another example, if theidentified terms are “what is” and is followed by two hundredmilliseconds, then the enhanced end of utterance detector 330 determinesthat “what is” matches three of the text samples in the text corpus 345and “what is” is the prefix of two thousand text samples in the textcorpus 345. In some implementations, the enhanced end of utterancedetector 330 uses the general endpointing signal from the generalendpointer 340 instead of identifying terms that are followed by a pausethat satisfies a threshold.

The prefix or complete utterance calculator 335 uses the determinedquantity of text samples to classify the identified terms as likely anincomplete utterance or not likely an incomplete utterance. The prefixor complete utterance calculator 335 computes a ratio using the numberof text samples in the text corpus 345 that match the identified termsand the number of text samples in the text corpus 345 that begin withthe identified terms. The prefix or complete utterance calculator 335compares the determined ratio to a threshold ratio. If the computedratio satisfies the threshold ratio, then the identified terms are notlikely an incomplete utterance. If the computed ratio does not satisfythe threshold ratio, then the identified terms are likely an incompleteutterance. For example, if the threshold ratio is 1:200 and theidentified terms are “what is,” then the determined ratio would be3:2000. In this instance, the computed ratio is less than the thresholdratio, so the prefix or complete utterance calculator 335 would classifythe identified terms as likely an incomplete utterance. As anotherexample, if the threshold ratio is 1:200 and the identified terms are“who is,” then the computed ratio would be 1:150. In this instance, thecomputed ratio is greater than the threshold ratio, so the prefix orcomplete utterance calculator 335 would classify the identified terms asnot likely an incomplete utterance.

In some implementations, once the prefix or complete utterancecalculator 335 classifies an utterance as not likely incomplete, theenhanced end of utterance detector 330 deactivates the microphone 315 sothat the audio subsystem 305 does not receive an additional audiosignal. In some implementations, once the prefix or complete utterancecalculator 335 classifies the a utterance is likely incomplete, theenhanced end of utterance detector 330 maintains the microphone 315 inan active state so that the audio subsystem 305 receives an additionalaudio signal.

The enhanced end of utterance detector 330 outputs the identified termsas a transcription or audio clip 355 once the enhanced end of utterancedetector 330 classifies the utterance as not likely incomplete. In someimplementations, the transcription or audio clip 355 is a transcriptionof the identified terms that the enhanced end of utterance detector 330classifies as not likely incomplete. For example, the transcription oraudio clip 355 may be the terms “what is the meaning of life.” In someimplementations, the transcription or audio clip 355 is an audio clip ofthe identified terms that the enhanced end of utterance detector 330classifies as not likely incomplete. For example, the transcription oraudio clip 355 may be an audio clip that corresponds to the term“whois.” The audio clip may be shorted to only include the portion ofthe audio signal 310 that contains the identified terms. The audio clipmay be accompanied with a transcription or additional metadata.

FIG. 3 is a diagram of an example process for classifying whether anutterance is likely incomplete. The process 400 may be performed by acomputing device such as the computing device 110 from FIG. 1A. Theprocess 400 analyzes audio data and classifies an utterance as likelyincomplete or not likely incomplete using word comparisons.

The computing device obtains a transcription of an utterance (410). Theutterance may be received from a user speaking into a microphone of thedevice and the device receiving an audio signal corresponding to theuser speaking. The device processes the audio signal to identify atranscription of the terms that the user is speaking. In someimplementations, the device may record timing data that specifies thetime between terms of the transcription.

The computing device determines, as a first value, a quantity of textsamples in a collection of text samples that (i) include terms thatmatch the transcription, and (ii) do not include any additional terms(420). The collection of text samples may be a collection of searchqueries that can be tied to a particular user or that are generated bymultiple users. In some implementations, the device identifies thequantity of text samples that include the terms in the same order as thetranscription. For example, the transcription is “what is.” The deviceidentifies five text samples in the collection of text samples thatinclude the terms “what” and “is” and no other terms. The deviceidentifies three of those text samples include “what” and “is” in thesame order as “what is.” In this example, the first value three.

The computing device determines, as a second value, a quantity of textsamples in the collection of text samples that (i) include terms thatmatch the transcription, and (ii) include one or more additional terms(430). In some implementations, the device identifies the quantity oftext samples that include the terms of the transcription as a prefix ofthe text samples. For example, the transcription is “what is.” Thedevice identifies three thousand text samples in the collection of textsamples that include the terms “what is” as a prefix. For example, “whatis the weather” includes the prefix “what is.” The device may alsoidentify fifty text samples in the collection of text samples thatinclude “what is” as an infix. For example, “most popular movie what isit” includes “what is” as an infix. The device may also identify twotext samples in the collection of text samples that include “what is” asa postfix. For example, “scary monster what is” includes “what is” as apostfix. In this example, the second value is three thousand.

The computing device classifies the utterance as a likely incompleteutterance or not a likely incomplete utterance based at least oncomparing the first value and the second value (440). In someimplementations, the device computes a ratio of the first value to thesecond value and compares the computed ratio to a threshold radio. Basedon the comparison of the computed ratio to the threshold ratio, thedevice may classify the utterance as likely incomplete. For example, ifthe computed ratio of the first value to the second value is 3:3000 andthe threshold ratio is 1:200, then with the computed ratio less than thethreshold ratio, the device classifies the utterance as likelyincomplete. If the computed ratio of the first value to the second valueis 1:150 and the threshold ratio is 1:200, then with the computed ratiogreater than the threshold ratio, the device classifies the utterance asnot likely incomplete.

In some implementations, the device classifies the utterance as likelyincomplete and maintains the microphone of the device in an active stateto receive an additional utterance. In some implementations, the deviceclassifies the utterance as not likely incomplete and deactivates themicrophone of the device and begins to process the utterance. In someimplementations, the device may classify the utterance as complete basedon the user pausing for a long time after speaking. For example, thedevice may conclude, by using a general endpointer, that “what is”followed by a long pause is a complete utterance. If the device compares“what is” to the collection of text samples and classifies “what is” aslikely an incomplete utterance, the device may override the initialconclusion that the utterance is complete. The device may maintain themicrophone in an active state to receive an additional utterance.

In some implementations, the device classifies the utterance as notlikely incomplete and processes the utterance. The device may submit atranscript of the utterance to a search engine or other natural languageprocessing engine. In some instances, the search engine or other naturallanguage processing engine may determine an appropriate response to theutterance and output the response to device for playback to the user.The search engine or other natural language processing engine maydetermine a response that has the possibility to elicit a reply from auser. In that instance, the search engine or other natural languageprocessing engine may include a flag in the response for the device tomaintain the microphone in an active state so that the user may reply.For example, the device submits an utterance “what time is it” to asearch engine and the search engine determines it is three o'clock. Inthis example, there is little likelihood that the user will have afollow on response to the time. Therefore, the search engine providesthe data to the device so that the device tells the user that the timeis “three o'clock” and the data includes a flag for the device not tomaintain the microphone in an active state after telling the user thetime. As another example, a device submits an utterance “when is thenext baseball game” to a search engine and the search engine determinesthat an appropriate response is “The baseball game is on Sunday. Wouldyou like to set a reminder?” In this example, there is a high likelihoodthat the user will have a follow on response to the question regardingsetting the reminder. Therefore, the search engine provides the data tothe device so that the device tells the user ““The baseball game is onSunday. Would you like to set a reminder?” and the data includes a flagfor the device to maintain the microphone in an active state aftertelling the user the time so that the user may speak the answer withoutmanually activating the microphone or using a hot word.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinnovations or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular innovations.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A computer-implemented method when executed ondata processing hardware causes the data processing hardware to performoperations comprising: receiving audio data corresponding to anutterance; processing, using a speech recognizer, the audio data togenerate a transcription of the utterance by recognizing each term inthe utterance; recording an amount of time after each term is recognizedby the speech recognizer; identifying an endpoint after one of the termsrecognized by the speech recognizer when the recorded amount of timeafter the term is recognized by the speech recognizer satisfies a firstthreshold before a subsequent term is recognized by the speechrecognizer; based on the endpoint identified after the one of the termsrecognized by the speech recognizer, determining whether the utteranceis one of likely complete or likely incomplete based on thetranscription of the utterance; and overriding the endpoint identifiedafter the one of the terms recognized by the speech recognizer when theutterance is likely incomplete.
 2. The method of claim 1, whereindetermining whether the utterance is one of likely complete or likelyincomplete comprises: comparing the transcription of the utterance to afirst collection of text samples identified as complete utterances;comparing the transcription of the utterance to a second collection oftext samples identified as incomplete utterances; and determiningwhether the utterance is one of likely complete or likely incompletebased on comparing the transcription of the utterance to the firstcollection of text samples identified as complete utterances andcomparing the transcription of the utterance to the second collection oftext samples identified as incomplete utterances.
 3. The method of claim2, wherein comparing the transcription of the utterance to the firstcollection of text samples identified as complete utterances comprises:determining a first quantity of text samples in the first collectionthat match the transcription of the utterance; and determining a secondquantity of text samples in the second collection that match thetranscription of the utterance.
 4. The method of claim 2, whereincomparing the transcription of the utterance to the first collection oftext samples identified as complete utterances comprises: determiningwhether terms in each text sample in the first collection occur in asame order as terms of the transcription of the utterance; anddetermining whether terms in each text sample in the second collectionoccur in a same order as terms of the transcription of the utterance. 5.The method of claim 1, wherein the operations further comprise, when theutterance is likely complete, determining to designate the endpointidentified after the one of the terms recognized by the speechrecognizer at an end of the audio data of the utterance.
 6. The methodof claim 5, wherein the operations further comprise, based ondetermining to designate the endpoint identified after the one of theterms recognized by the speech recognizer at the end of the audio dataof the utterance, deactivating a microphone that detected the utterance.7. The method of claim 1, wherein overriding the endpoint identifiedafter the one of the terms recognized by the speech recognizer comprisesmaintaining a microphone that detected the utterance in an active state.8. The method of claim 7, wherein maintaining the microphone thatdetected the utterance in the active state permits the speech recognizerto process additional audio data.
 9. The method of claim 1, wherein thespeech recognizer resides on a user device.
 10. The method of claim 1,wherein the speech recognizer resides on a server.
 11. A systemcomprising: data processing hardware; and memory hardware incommunication with the data processing hardware and storing instructionsthat when executed on the data processing hardware causes the dataprocessing hardware to perform operations comprising: receiving audiodata corresponding to an utterance; processing, using a speechrecognizer, the audio data to generate a transcription of the utteranceby recognizing each term in the utterance; recording an amount of timeafter each term is recognized by the speech recognizer; identifying anendpoint after one of the terms recognized by the speech recognizer whenthe recorded amount of time after the term is recognized by the speechrecognizer satisfies a first threshold before a subsequent term isrecognized by the speech recognizer; based on the endpoint identifiedafter the one of the terms recognized by the speech recognizer,determining whether the utterance is one of likely complete or likelyincomplete based on the transcription of the utterance; and overridingthe endpoint identified after the one of the terms recognized by thespeech recognizer when the utterance is likely incomplete.
 12. Thesystem of claim 11, wherein determining whether the utterance is one oflikely complete or likely incomplete comprises: comparing thetranscription of the utterance to a first collection of text samplesidentified as complete utterances; comparing the transcription of theutterance to a second collection of text samples identified asincomplete utterances; and determining whether the utterance is one oflikely complete or likely incomplete based on comparing thetranscription of the utterance to the first collection of text samplesidentified as complete utterances and comparing the transcription of theutterance to the second collection of text samples identified asincomplete utterances.
 13. The system of claim 12, wherein comparing thetranscription of the utterance to the first collection of text samplesidentified as complete utterances comprises: determining a firstquantity of text samples in the first collection that match thetranscription of the utterance; and determining a second quantity oftext samples in the second collection that match the transcription ofthe utterance.
 14. The system of claim 12, wherein comparing thetranscription of the utterance to the first collection of text samplesidentified as complete utterances comprises: determining whether termsin each text sample in the first collection occur in a same order asterms of the transcription of the utterance; and determining whetherterms in each text sample in the second collection occur in a same orderas terms of the transcription of the utterance.
 15. The system of claim11, wherein the operations further comprise, when the utterance islikely complete, determining to designate the endpoint identified afterthe one of the terms recognized by the speech recognizer at an end ofthe audio data of the utterance.
 16. The system of claim 15, wherein theoperations further comprise, based on determining to designate theendpoint identified after the one of the terms recognized by the speechrecognizer at the end of the audio data of the utterance, deactivating amicrophone that detected the utterance.
 17. The system of claim 11,wherein overriding the endpoint identified after the one of the termsrecognized by the speech recognizer comprises maintaining a microphonethat detected the utterance in an active state.
 18. The system of claim17, wherein maintaining the microphone that detected the utterance inthe active state permits the speech recognizer to process additionalaudio data.
 19. The system of claim 11, wherein the speech recognizerresides on a user device.
 20. The system of claim 11, wherein the speechrecognizer resides on a server.